The perfect combination: GenAI and persuasion strategies for unbeatable A/B tests, Part 2

In my last post, I covered how to write prompts for generative AI to create an email campaign brief that incorporates persuasion principles.

This article shares the results that ChatGPT delivered using my extensive prompt, which I wrote to align with the hypothesis we used to set up our A/B testing program. 

Some notes before we dive in:

• We wanted to learn how ChatGPT could factor into our campaign process at all stages to help us set up, test and launch campaigns faster. So, we didn’t send this campaign to our customers and I won’t have data on which version drove better results. But we do have a better idea now for how to use ChatGPT more effectively in the creation process and we’ll use that knowledge in our next campaign.
• We used ChatGPT as our language bot in these trials. But everything we have said applies no matter which bot you use. You could even test several tools to see if the same prompt yields different results depending on the bot you choose.

Did ChatGPT deliver what we wanted?

Sort of. I was impressed with the depth of material that our prompt produced and how well the results stuck to the prompt, more or less. None of the copy was anywhere near ready for testing. But it did save us hours of content creation and gave us a starting point for refining and polishing. 

The prompt

Here’s the prompt we created for this campaign, intended to promote a summer dress sale for an ecommerce fashion retailer:

“You’re an email marketer for an ecommerce clothing retailer tasked with writing compelling copy to increase conversions for an email campaign. You’re selling a new range of summer sandals that are they are bright and colorful, classy and comfortable.

Deliver one subject line, opening paragraph and call to action, using each of these persuasion principles and cognitive biases below, without actually mentioning the cognitive bias within the copy, but label each example with the principle or bias:

Reciprocity, scarcity, social proof, authority, anchoring, loss aversion, curiosity gap, endowment effect, confirmation bias, emotional appeal.”

The boldface copy lists the 10 persuasion principles or cognitive biases we wanted to compare to learn which would generate more activity and, ultimately, more sales, because that was the goal for this sample campaign.

The hypothesis for the test

The prompt aligns with this hypothesis: “Emotional appeals will generate more conversions than social proof.” 

Why this match-up? Because we expect to learn that emotion will generate more conversions than social proof because evocative copy will be more appealing.

A hypothesis is essential here because we are testing more than Subject Line 1 versus Subject Line 2 or Call to Action A against CTA B and we will be testing this multiple times across other campaigns. This is a key element of testing using holistic email marketing, a practice I developed that goes beyond testing individual elements to consider customer emotions and motivations. 

Starting with a hypothesis assures that every element of the email message, from the subject line to the CTA, lines up with this testing concept and that makes your testing results more reliable and more applicable beyond email to web copy, social media and beyond. 

The results

To save space, I’ll review only the unedited campaign copy from the two comparison principles: emotional appeal and social proof.

Emotional appeal: We chose this because emotions so often guide people’s actions.

• Subject line: Step into Summer Bliss: Embrace the Joy of Our Sandals!  
• Copy: Hi there, Sunshine Seeker! With the arrival of summer, it’s time to embrace the joy, freedom and comfort that the season brings. Our new range of sandals isn’t just about style – it’s about evoking the emotions of carefree days, warm evenings and endless adventures. Step into the magic of the season with sandals that resonate with your heart and soul.
• CTA: Experience Summer Bliss with Our Sandals!

Social proof: We chose this “wisdom of crowds” approach because we are social animals and we like to see what others are doing and to get support for our own decisions.

• Subject line: Join the Sandal Craze: Shop Best-Sellers of the Season!
• Copy: Hi there, Fashion Insider! It’s time to put your best foot forward with our bestselling summer sandals that have captured the hearts of countless style enthusiasts. These are the pairs that everyone’s talking about – the ones that are transforming everyday outfits into unforgettable moments. Join the league of satisfied customers who’ve embraced the comfort and style of our sandals. Step into the limelight with confidence!
• CTA: Shop Sandals Loved by Hundreds!

And here’s how mockups of those two email messages compare side by side:

Emotional appeal	Social proof
Step into Summer Bliss: Embrace the Joy of Our Sandals!	Join the Sandal Craze: Shop Best-Sellers of the Season!

Our thoughts

As I mentioned earlier, ChatGPT’s raw copy is not ready for prime time. I would tweak everything, beginning with the unnecessarily long emotional-appeal subject line. The copy is also too on-the-nose as it mentions “evoking the emotions of” even though we told ChatGPT not to do that. 

The social proof version is somewhat better although again I would polish the subject line and tighten up the copy. But ChatGPT has already saved us a good hour or more in delivering a serviceable first draft. 

That’s essential to remember as you incorporate GenAI into your everyday email operations, especially with testing. You can direct it to deliver 10, 20 or however many versions of a subject line to consider for testing as long as you give it precise instructions in your prompt about what they should or should not include.

Adding ChatGPT to a six-step testing and optimization flow

1. Create the hypothesis 

You can use ChatGPT to suggest a hypothesis that covers the variables you want to compare and the results you expect to see. The variables would cover entire effect of each version of the campaign instead of focusing on a single element like a subject line. Because this is a more complex hypothesis, having ChatGPT deliver a rough draft can inspire you to create a viable second draft.

2. Develop the campaign 

This step includes all of the campaign preparation you normally include, beginning with a detailed campaign brief for the two versions (the A and B versions) you plan to test that lays out: 

• The goal.
• Content elements.
• Visuals.
• Call to action.
• Expected other activities.
• Copy tone.

Be sure all elements of the campaign line up with your testing hypothesis so you get the most viable results.

Use ChatGPT to develop some or all of the copy for your campaign. If you are not using branded images like models or product shots, you can experiment with DALL-E, the genAI image creator. Write a detailed prompt like the one I shared above (and see Part 1 of this post to discover the eight points I used to create that prompt).

Dig deeper: The new frontier of visual content: A marketer’s guide to AI

3. Run the test

Track the test using your chosen KPI, such as conversions for an ecommerce campaign. Test using a 50/50 split and let your campaign run for a minimum of seven days to get the best chance of reaching statistical significance — the point at which differences emerge that are more likely to result from your testing variations than from random events. You can wait even longer depending on your audience or customer behavior.

4. Declare a winner if you can

But don’t stop there. Set up another test that follows the same hypothesis but uses a different offer and product(s). Review your findings and run another test. Can you pick a winner in two out of three tries? 

This will reveal whether your customers follow the same behavior even in different circumstances. That’s information you can share in the last step of this testing process. (See step 6 below). 

5. Record your results

Be as specific as you can. Record all of the copy elements, everything in your campaign brief and your testing notes, including your hypothesis and writing prompts. Let everyone on your team see the results and consider printing them out and compiling them in a research notebook. 

6. Share what you learned

Your email customers are like one giant focus group. They’re a window into your wider database. You might be able to apply what you learn about them here across your other marketing channels. Testing might also reveal stark differences in how people behave on the website or social media compared with email.

This is valuable information that can help your company know its customers better. So don’t hide your learnings in a folder! Share them with other teams in the marketing department and beyond.

Dig deeper: 7 common problems that derail A/B/n email testing success

Key takeaways on persuasion, ChatGPT, email and testing 

Use persuasion principles and cognitive bias to help your audience act

Your customers are on your email list because they have a reason to seek you out. If you can help them solve problems, achieve goals or just enjoy hearing from you, they will help you out by buying from you. That’s helpful marketing in a nutshell.

Language bots are still evolving

We’re still in the early stages of development and more new tools that use AI in general or generative AI in particular are coming online soon. They will always need human oversight to make them work best for you. You bring knowledge of your products, marketing goals, branding devices and audiences they don’t supply. 

Let these tools do the hard job of going from a blank screen or notes on a whiteboard to the first draft. I find it easier to edit than to start from scratch, don’t you?

Be cautious when using persuasion principles and cognitive bias

Each one has a specific use and appeals to a limited part of your audience. Loss aversion is popular because nobody wants to lose out on a good deal or a way to fix a problem once and for all. But a steady diet of loss aversion can introduce the email equivalent of banner blindness. 

People just stop responding to all those “Don’t miss out!” entreaties. They know another deal is coming. Your campaign strategy should include a plan and reasoning for taking a specific angle. Use them to find a different way to make them care about the current deal. 

Help your ChatGPT get smarter 

Most chatbots like ChatGPT will allow you to write multiple prompts in a single session and then come back and pick up where you left off if you have to break off the session. 

When you keep all of your prompting requests in the same session, the bot can learn from all of those iterations and adaptions and deliver sharper copy in follow-up rounds.

When you’re writing prompts on a similar kind of product, choose the option that lets you reopen an earlier, related session. If you start fresh each time, you’re making more work for yourself by repeating the same prompts. 

Keep proprietary information out of your prompts

Yes, ChatGPT has a big mouth. Anything you include in a prompt becomes fair game for anyone asking for something similar. That cooperation can help you in some ways but can also hurt you if you add private product information, audience characteristics, personal data and other sensitive info.

Don’t let an apparent failure turn you off regular testing

It happens to everyone. If your test doesn’t support your hypothesis, use what you learned from this test to refine your efforts and try again. Recording your failures as well as your wins.

But also – keep testing! Maybe you needed to let a test run longer. You might go back in a couple of days and discover your results have changed.

Keep trying new things

Your audience and customers are always changing, ever so slightly. A regular testing program helps you discover these changes and evolve your marketing approaches to respond to them. 

Dig deeper: How persuasive email design can influence the ecommerce customer journey